System for interaction between an autonomous vehicle and a vulnerable user

ABSTRACT

A system for implementing an interaction between a semi-autonomous or autonomous motor vehicle and a vulnerable user of a carriageway used by the motor vehicle and potentially crossed by a vulnerable user at a crossing, pedestrians and cyclists being considered to be vulnerable users. The system including an environment sensor, a computer processing the data received from the one or more environment sensors, and a display device arranged on the vehicle and directed to the outside of the vehicle, the display device in the form of a horizontal strip and occupying a longitudinal end of the vehicle, and providing vulnerable users present in the vehicle&#39;s environment in the vicinity of the crossing with a visual indication as to whether or not they may cross the carriageway road surface, so that pedestrians and cyclists can obtain unambiguous information as to their presence being taken into account by the vehicle.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is the U.S. National Phase Application of PCTInternational Application No. PCT/EP2020/074614, filed Sep. 3, 2020,which claims priority to French Patent Application No. 1909995, filedSep. 11, 2019, the contents of such applications being incorporated byreference herein.

FIELD OF THE INVENTION

The present invention relates to systems for interaction between anautonomous vehicle (or semi-autonomous or partially autonomous vehicle)and one or more road users such as, for example, a pedestrian orcyclist.

BACKGROUND OF THE INVENTION

Autonomous or semi-autonomous vehicles have to travel on carriageways ineither its own or a public area.

In both cases, these carriageways have protected crossings forpedestrians and other users, referred to as “vulnerable users”.Vulnerable users include cyclists, pedestrians, adults or children orold people, users of various strollers, scooters, skateboards, and otherpersonal mobility devices, monocycles, etc.

The question that arises for a vulnerable user is that of determiningwhether they have been perceived and taken into account by theautonomous vehicle, and relatedly whether the autonomous vehicle willstop and let said vulnerable user cross when the latter is about tocross the road.

A need has arisen to provide a solution so that the autonomous vehiclecan give a clear, intuitive and unambiguous message to one or morevulnerable users which reflects the decisions made by the vehicle'sonboard electronics (akin to the intentions of a conventional driver).

SUMMARY OF THE INVENTION

To that end, what is proposed therefore is a system for implementing aninteraction between a semi-autonomous or autonomous motor vehicle and atleast one vulnerable user of a carriageway used by the motor vehicle andpotentially crossed by a vulnerable user at a crossing, which might ormight not be marked, pedestrians and cyclists being considered to bevulnerable users, said system comprising:

-   -   at least one environment sensor arranged on the vehicle,    -   at least one computer on board the vehicle, processing the data        received from the at least one environment sensor,    -   at least one display device arranged on the vehicle, and        directed to the outside of the vehicle, and controlled by the        computer,    -   the display device being in the form of a horizontal strip and        occupying at least one longitudinal end of the vehicle;    -   the display device comprising a matrix panel having at least        four pixels per cm²,    -   the display device being configured to provide vulnerable users        present in the vehicle's environment in the vicinity of the        crossing with at least one visual indication as to whether or        not they may cross the carriageway road surface, and delivers        either an indication prompting them to cross or, conversely, an        indication forbidding them from crossing, so that pedestrians        and cyclists can obtain unambiguous information as to their        presence being taken into account by the semi-autonomous or        autonomous motor vehicle.

Based on the analysis of the environment and on the presence ofvulnerable users, the autonomous vehicle decides, as applicable, to stopand give way or to continue and not to give way.

Advantageously, by analogy with the signals for pedestrians atconventional protected crossings, the autonomous or semi-autonomousvehicle provides vulnerable users present in the vehicle's environmentin the vicinity with an indication prompting them to cross or,conversely, an indication forbidding them from crossing, making use inparticular, but not exclusively, of the conventional colors (green, red,etc.). The level of information delivered to the vulnerable user can bemore or less elaborate, as will be seen below. However, it will be notedthat the information rendered on the display device differs from anoptical facsimile of the scene. Thus, a vulnerable user is aware of theautonomous or semi-autonomous vehicle's intentions and can cross theroad calmly in front of said vehicle.

In various embodiments of the invention, one or more of the followingarrangements or features may also be used, taken individually or incombination.

According to one aspect, the display device is configured to display arepresentation of the scene comprising the protected crossing, with theusers symbolically represented there by character avatars which may bestationary or moving. As a result, the vulnerable user identifies thatthe autonomous vehicle has indeed analyzed and “understood” thesituation as it is and has taken the presence of these one or morevulnerable users into account. Such a symbolic representation of thescene improves the intelligibility of the message intended for thevulnerable user, in particular with reference to the learning mechanismsthat take place in the subject's childhood, with acquisition of theidentity of the characters including their own, and matching of theimages as perceived in a mirror with images perceived directly (whileaccounting for symmetry). The representation of the scene by the displaydevice reproduces this mirror effect after a fashion, which makes thedisplayed message intuitive and easy to understand.

According to one aspect, the rendering, by the display device, of thepresence of one or more vulnerable users present in the vehicle'senvironment in the vicinity of the protected crossing is performed inreal time and the respective positions of avatars representing the usersrelative to the protected crossing reflect reality with a delay of lessthan 1 second, preferably less than 600 ms.

It is indeed a depiction of the actual scene that is represented andthat, being perceived in real time by the vulnerable user, makes it moreintuitive to grasp that what is taken into account by the vehicle'sonboard electronics is correct and corresponds to the actual situationexperienced by the vulnerable user. This strengthens the intelligibilityof the message.

According to one aspect, provision is made in the animation rendered bythe display device in real time for the number of avatars to correspondto the number of people actually detected in the environment.

This strengthens the representativeness of the rendering on the displaydevice with respect to the real world.

Thus, it is easier for the vulnerable user to recognize that it isindeed the scene actually experienced which is symbolically representedon the vehicle's display.

According to one aspect, the strip can occupy substantially the entirewidth of the front face of the vehicle and/or the entire width of therear face of the vehicle. As a result, on one hand there is a large areaavailable for displaying an explicit message and on the other hand thismessage can be picked up by vulnerable users who are located on eitherside of the vehicle.

It is not excluded for the display strip to continue laterally over tothe lateral portion of the vehicle on each side; this maximizes thevisibility of the display to vulnerable users nearby.

According to one aspect, the vehicle can comprise a display device onits front face and on its rear face, preferably over the entire width.Advantageously, for a vehicle having a preferred direction of travel,the display device on the rear face will be intended for vehiclesfollowing it to indicate, for example, that overtaking is not advised,or even forbidden, because of vulnerable parties crossing the road infront of the autonomous vehicle. Furthermore, for two-way shuttle-typevehicles, when the shuttle vehicle changes the main direction of travel,the role of the front and rear displays is then reversed, and the twodisplays are used symmetrically.

According to one aspect, the computer is configured to classify theentities perceived by the one or more environment sensors. Thisclassification makes it possible to identify and isolate human or animalcharacters; said classification makes it possible, conversely, to removestreet furniture elements such as garbage cans, posts, barriers, as wellas vegetation elements such as shrubs or the like, etc.

Thus, the representation of the scene makes it possible to highlight thecharacters in particular and to partially or completely ignore fixedobjects which have no intention of crossing the road.

In addition, the representation of the scene can highlight, by means ofa differentiated representation, the classification of the variousentities, for example pedestrians, cyclists, scooter users, strollerusers as well as possibly a representation of the size of thecharacters, for example adults versus children. It is also not excludedfor the gender of the characters to be distinguished: female, male oreven neutral.

The avatars reflect each category of entity and contribute to realisticand intelligible rendering of the display, without this being an opticalfacsimile of the scene of interest.

According to another optional aspect, each avatar reflecting the type ofvulnerable user according to the classification can have characteristicsrepresentative of the classification of the entity: size of thecharacter, direction of movement, stationary posture, low speed, highspeed. Accessories used by the character can also be depicted; forexample a bicycle, a skateboard, a stroller, a scooter, a pet, etc.

According to another optional aspect, each avatar reflecting eachvulnerable user present in the scene is represented substantially at thegeographical position where they are located over time and in real time.The position of pedestrians, and more generally of vulnerable users,over time in the representation of the scene of interest letspedestrians easily understand that the autonomous vehicle understandsand monitors the situation as it changes in real time.

According to one aspect, the display device is arranged at a heightdefined by a bottom line of the display located above 1 m from theground and its top line located below 1.40 m from the ground.

In this way, an attempt is made to get as close as possible to thevisual interaction that occurs between a driver of a conventionalvehicle and a pedestrian in a case of a conventional situation of avehicle driven by a person. This arrangement at a position above theground corresponding to the area of a conventional driver's face allowsthe vulnerable user to move in a mixed-traffic environment with bothconventional vehicles driven by drivers and semi-autonomous orautonomous vehicles with or without a driver present.

In other words, the display device is located within the range [1 m-1.4m]; it is noted that this facilitates reading for the average user andprovides an analogy with the conventional case (personified driver).

In one aspect, the display device is configured to display an indicationthat the vehicle is about to start off and drive over the protectedcrossing.

This forms a clear and unambiguous warning/notification of the imminentmovement of the vehicle.

According to one optional aspect, the environment sensor arranged on thevehicle is configured to sense the environment on both sides of thecarriageway road surface to the front and thus react according to thepossible presence of vulnerable users on either side of the road. Inother words, the treatment can be advantageously symmetrical withrespect to both sides of the roadway.

According to one optional aspect, the vehicle can further comprise asound source, which is used in addition to the display device and iscontrolled by the computer.

According to one optional aspect, the vehicle can provide illuminationof the crossing, by means of lamps for example, which may bedirectional, as applicable, which may prove useful at night and whichstrengthens the perception and the intelligibility of the message thatthe vulnerable user receives from the autonomous vehicle.

This strengthens the visual message, which is of some use in particularfor vulnerable users focusing all of their attention on using theirsmartphone for example.

An aspect of the invention also relates to a method for interactionbetween a semi-autonomous or autonomous motor vehicle and at least onevulnerable user of a carriageway used by the motor vehicle andpotentially crossed by a vulnerable user at a crossing, which might ormight not be marked, pedestrians and cyclists being considered to bevulnerable users, said vehicle comprising at least one environmentsensor arranged on the vehicle, at least one computer on board thevehicle, at least one display device arranged on the vehicle, anddirected to the outside of the vehicle, and controlled by the computer,the method making provision for:

the environment sensor to detect entities present in the vehicle'senvironment and in particular in a region located in its direction oftravel,

the computer to classify the entities by identifying vulnerable usersincluding pedestrians and cyclists and users of scooters, skateboards,or strollers,

the display device providing vulnerable users present in the vehicle'senvironment in the vicinity of the crossing with at least one visualindication as to whether or not they may cross the carriageway roadsurface, and delivers either an indication prompting them to cross or,conversely, an indication forbidding them from crossing.

Thus pedestrians and cyclists can obtain unambiguous information as totheir presence being taken into account by the semi-autonomous orautonomous motor vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features, details and advantages will become apparent from readingthe following detailed description and from examining the appendeddrawings, in which:

FIG. 1 is a general basic diagram of one embodiment of the invention,

FIG. 2 shows one example of a basic diagram relating to the main displayscreen,

FIG. 3 shows a functional schematic view of the main display screen,

FIG. 4 illustrates two exemplary indications for allowing crossing or,conversely, forbidding crossing,

FIGS. 5A and 5B illustrate the display configuration on the strip of thevehicle's front display device, with pedestrians crossing in front ofthe vehicle,

FIG. 6 illustrates various types of avatars representing pedestrians,the avatars being inserted into the representation of the scene withtheir position and the change in each position over time,

FIG. 7 illustrates the display toward the end of the crossing sequenceindicating that the vehicle is about to start off, and

FIG. 8 illustrates one particular variant embodiment, relating to ashuttle-type autonomous vehicle.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the various figures, the same references have been used to referenceidentical or similar elements.

With reference to FIG. 1 , a semi-autonomous or autonomous motor vehicleVh1 travels on a carriageway 9 which can be either a road or a street.FIG. 1 shows a pedestrian crossing referenced by 90, which in thepresent case is marked by lines painted on the ground (alternation ofblack and white strips 92, 93). However, it should be noted that in somecases, there may be pedestrian crossings which are not marked on theground or else they are marked in another way, for example via crossingmarkings, pseudo crossing markings, lines and/or dots. In certain casesthe crossing may be qualified as a “protected crossing”, but theinvention is not limited to that case.

Regarding the vehicle, it is an autonomous, or semi-autonomous orpartially autonomous vehicle, that is to say that, in practice, this maycorrespond to a case where there is no one driving the vehicle, nor anyperson in any driving position (completely autonomous vehicle), this canalso correspond to a case where there is someone in the driving positionbut who delegates driving functions to the driver assistance functionsperformed by the onboard electronics. It is noted that the level ofdelegation can change over time.

It is noted that an aspect of the invention can also be applied to avehicle driven in a conventional manner but with tinted windows thatmake visual contact between pedestrians and the driver impossible.

It should be noted that the vehicle in question can be a privatevehicle, a minibus, a van, a delivery truck, a taxi, or a self-drivingtaxi. The vehicle in question can also be a tram, a trolleybus, atram-train, etc.

The vehicle in question can be designed to travel on carriageways ineither its own or a public area, or in mixed situations where it usescarriageways in its own area and then carriageways in a public area orvice versa.

As will be seen in more detail below, the vehicle Vh1 is provided withone display device 1 at the front of the vehicle (in the normaldirection of travel) and, optionally, another display device 101 locatedat the rear of the vehicle.

The carriageway can be one-way or can be two-way. It is not excluded forthere to be a plurality of parallel lanes in the same direction whichcorresponds, in everyday language, to a multi-lane traffic thoroughfare.

Coming now to the users of the carriageway, “vulnerable users” aregenerally defined as those people present who are not in a vehicle'spassenger compartment; the concept of “vulnerable users” includespedestrians, cyclists, adults, children or old people, users of variousstrollers, scooters, skateboards, personal mobility devices, monocycles,etc. The concept of “vulnerable users” can also include pets.

FIG. 1 shows a pedestrian U1 located to the left of the road from thevehicle's point of view, a cyclist U2 preparing to cross the road fromthe left, and an adult U4 and a child U3 who are both pedestrians andare preparing to cross the road from the right from the vehicle's pointof view.

The display device 1 can be presented as illustrated in FIG. 2 with amain strip 11 which extends over the entire width of the vehicle.

In the example illustrated, the display device 1 is rectangular with abottom edge 1 a and a top edge 1 d.

The display device is arranged at least on one longitudinal end of thevehicle.

A lateral extension 12 onto the side of the vehicle can be provided,either on one side only or on both sides symmetrically.

The strip is horizontal, that is to say wider than it is high. It may begenerally rectangular in shape.

The typical dimensions of the horizontal strip are at least 1.60 m inwidth and 30 cm in height. Generally, the height H1 will be within arange [20 cm-40 cm]. Generally, the width L1 will be within a range [140cm-200 cm].

The lateral extension 12 may have the same height, or even a greaterheight, and it may have a length in the longitudinal direction ofbetween 20 and 50 centimeters.

The height position of the display device is defined by a bottom line ofthe display located above 0.9 m from the ground and its top line locatedbelow 1.50 m from the ground. Stated otherwise, the display device islocated within the range [0.9 m-1.5 m].

The display device comprises a matrix panel whose resolution is at leastfour pixels per cm², or 25 pixels per square inch. Of course, a higherresolution is entirely possible.

The display device can be in the form of an LED matrix, either a matrixof multicolor LEDs or interlaced matrices of monochromatic LEDs. Theprimary colors can be used as the basis for generating, in particular,the colors red and green but also other, additional colors.

The vehicle is equipped with one or more environment sensors 3.Regarding the environment sensor, it can be a lidar sensor, for examplewith 360° coverage; it can be a radar; it can be a 3D camera; it can bea time-of-flight (TOF) camera; or it can be a combination of a pluralityof sensors.

In addition, provision is made to use an infrared camera which makes itpossible to detect the temperature of each source and, in particular,facilitates the classification of entities in order to extract livingbeings from the environment.

FIG. 1 shows, for example, a central sensor 3 arranged on the roof ofthe vehicle, for example a 360° lidar. In addition, another sensor 31 isprovided at the front of the vehicle chosen from the technologies listedabove. In addition, another sensor 32 is provided at the rear of thevehicle chosen from the technologies listed above.

According to another configuration illustrated in FIG. 8 , provision canbe made for a sensor 3 located in the upper region of the vehicle'sfront portion, and sensors 31, 32 arranged in the vehicle's bumper, eachof these sensors being able to be of any of the above types.

The vehicle comprises at least one computer referenced by 4; saidcomputer may also be referred to as an “electronic control unit”(“ECU”).

The computer 4 processes the data received from the one or moreenvironment sensors and performs one or more operations for classifyingthe entities that are detected in the images or, more generally, thedata received from the sensors.

In particular, an algorithm is provided which makes it possible toextract, from the data or images received from the sensors, entitiescorresponding to living beings, people or animals in the vehicle'simmediate environment.

The classification of the entities may involve image content analysisand/or spectral analysis.

Certain elements can give rise to easier recognition, such as bicycle orstroller wheels; specifically, a continuous or discontinuous ellipse (aplurality of arcs of an ellipse) is recognized.

The classification of the entities can also make use of a motionanalysis, in particular to identify the motions of people in the data orimages received from the sensors.

After analyzing the data received from the sensors, if it is determinedthat one or more vulnerable users are about to cross the roadway, thecomputer can decide to stop the autonomous vehicle and give way to allowthe vulnerable users to cross.

It is noted that analysis and classification are performed continuouslyunder the conditions where the vehicle has decided to stop to allowpedestrians to cross.

It is noted that analysis and classification continue as vulnerableusers cross the road.

The display device 1 is configured to provide the vulnerable userspresent in the vehicle's environment in the vicinity of the protectedcrossing with at least one visual indication as to whether or not theymay cross the carriageway road surface, and delivers either anindication prompting them to cross or, conversely, an indicationforbidding them from crossing.

Particularly, the visual indication may use a conventional color code,namely green to indicate a prompt to cross and red to indicate thatcrossing is forbidden.

The visual indication can, in particular, use all or some of thepictograms commonly used at pedestrian crossings, namely a pictogramindicating a prompt to cross in green, illustrated by reference 21 inFIG. 4 , and a pictogram indicating that crossing is forbidden in red,illustrated by reference 22 in FIG. 4 . The color orange can also beused for the pictograms. Using effects involving changes of color fromamong the above colors or else other colors is not excluded either.

The conventional pictograms 21, 22 can, in particular, be displayed onthe lateral extensions 12 of the display device, as by analogy with thepictograms for crossings for pedestrians.

The display is updated in real time, according to the stream ofinformation received from the environment sensors.

The display device is configured to display a representation of thescene comprising the protected crossing, as illustrated in FIGS. 5A, 5Band 7 , with the users symbolically represented there by characteravatars which may be stationary or moving.

Each avatar reflecting the type of vulnerable user according to theclassification can have characteristics representative of theclassification of the entity: size of the character, direction ofmovement, stationary posture, low speed, high speed. Accessories used bythe character can also be depicted; for example a bicycle, a skateboard,a stroller, a scooter, a pet, etc.

More precisely, as illustrated in FIG. 6 , the posture and size of theavatars represented reflects the presence and the posture of the peopleactually present in the vicinity of the vehicle, for example an adult ina stationary posture 51, a child in a stationary posture 52, two adultscrossing together 53, 54, and a walking teenager 55.

In FIGS. 5A and 5B, the character avatars are in the shape of cylinders50.

The computer 4 can access map databases which allow it to determine theposition of the protected crossings listed, without this constituting anexhaustive list however. Thus, the level of filtering performed by thecomputer with regard to the human or animal entities classified maydepend on the presence or absence of a listed protected crossing; inpractice, the decision to stop to allow pedestrians to cross will befavored at the protected crossings listed.

In the event that no pedestrian crosses despite the presence of entitiesclassified as persons at the sides of the road, the autonomous vehicleswitches back from the message prompting crossing to a messageforbidding crossing and/or a message indicating that the vehicle isabout to start off.

As shown in FIG. 3 , the computer 4 receives a continuous stream of datafrom the one or more environment sensors 3, 31, 32 via an onboard localarea network or wirelessly, and it is connected to one or more displaysvia an onboard local area network or wirelessly.

In addition, a sound source can be provided, which is used in additionto the display device and is controlled by the computer, to make itpossible to strengthen the level of information for the users,complementing purely visual information which might be insufficient forcertain vulnerable users focused on using their smartphone.

In addition, a light source can be provided on the vehicle, used inaddition to the display device, and controlled by the computer 4, toallow the crossing, which might or might not be marked on the ground, tobe illuminated, in order to strengthen the perception andintelligibility of the message given by the display device to avulnerable user crossing, or about to cross, the road.

It is noted that the semi-autonomous or autonomous vehicle, when itstops to allow pedestrians to cross, observes a safety distance,referenced by DG, with respect to the marked crossing, as illustrated inFIGS. 5A and 5B. The safety distance DG can be configured to be between1 m and 3 m; the safety distance can also be dependent on thecircumstances, for example day or night, the type of carriageway, or theclassification of the vulnerable users identified.

With reference to FIG. 8 , the front façade of the vehicle is almostvertical with, from bottom to top, a bumper in the bottom portion, thena front body region, then the main display strip 11, then a glazedregion 18, above which is one of the sensors 3.

It is noted that the lateral extensions 12 on the side of the vehiclehave a position that is slightly higher than the front central strip 11.For example, the lateral extensions 12 each extend in the longitudinaldirection over at least 30 cm, and over a height of 25 to 40 cm.

In the example shown, the display device is continuous, i.e. there is nointerruption between the lateral extensions 12 and the front centralstrip 11, which allows an avatar to move continuously from one side tothe other without any singular point, as when the vulnerable user iscrossing. However, it is not excluded for there to be a front strip andlateral displays which are not directly an extension of the front strip.

The invention claimed is:
 1. A system for implementing an interactionbetween a semi-autonomous or autonomous motor vehicle and at least onevulnerable user of a carriageway used by the motor vehicle andpotentially crossed by a vulnerable user at a crossing, which might ormight not be marked, pedestrians and cyclists being considered to bevulnerable users, said system comprising: at least one environmentsensor arranged on the vehicle; at least one computer on board thevehicle, processing the data received from the at least one environmentsensor; and at least one display device arranged on the vehicle, anddirected to the outside of the vehicle, and controlled by the computer,wherein, the display device is in the form of a horizontal strip andoccupies at least one longitudinal end of the vehicle; the displaydevice comprises a matrix panel having at least four pixels per cm², andthe display device being configured to provide vulnerable users presentin an environment of the vehicle in the vicinity of the crossing with atleast one visual indication as to whether or not they may cross thecarriageway road surface, and delivers either an indication promptingthem to cross if the vulnerable user is at a protected crossing or,conversely, an indication forbidding them from crossing if thevulnerable user is not at a protected crossing, so that pedestrians andcyclists can obtain unambiguous information as to their presence beingtaken into account by the semi-autonomous or autonomous motor vehicle,the display device also being configured to display a representation ofa scene comprising the crossing, the users being symbolicallyrepresented by character avatars which may be stationary or moving, ofwhich the rendering, by the display device, is performed in real timeand of which the respective positions relative to the crossing reflectreality with a delay of less than 1 second, and of which a numbercorresponds to the number of people actually detected in the vehicle'senvironment.
 2. The system as claimed in claim 1, wherein the stripoccupies substantially an entire width of a front face of the vehicleand/or an entire width of a rear face of the vehicle.
 3. The system asclaimed in claim 1, wherein the vehicle comprises a display device onits front face and on its rear face, over the entire width.
 4. Thesystem as claimed in claim 1, wherein the computer is configured toclassify entities perceived by the one or more environment sensors. 5.The system as claimed in claim 1, wherein the display device is arrangedat a height defined by a bottom line of the display located above 1 mfrom the ground and its top line located below 1.40 m from the ground.6. The system as claimed in claim 1, wherein the display device isconfigured to display an indication that the vehicle is about to startoff and drive over the crossing.
 7. A vehicle comprising a system asclaimed in claim
 1. 8. The system as claimed in claim 2, wherein thevehicle comprises a display device on its front face and on its rearface, over the entire width.
 9. The system as claimed in claim 1,wherein the delay is less than 600 ms.
 10. The system as claimed inclaim 1, wherein the computer obtains information from a map database todetermine if the crossing is a protected crossing.